Fix renderer OOM crash loop during large collection installs - #23905
Fix renderer OOM crash loop during large collection installs#23905DoomerDGR8 wants to merge 3 commits into
Conversation
Installing a very large collection (e.g. 2852 mods) against a big archive library crashed the renderer with reason "oom" shortly after dependency resolution, looping forever on resume. The pipeline ran several O(refs x downloads/mods) scans that allocated fresh lookup objects per probe (~8.5M probes per pass), retained full metadb result lists per dependency, re-scanned all installed mods on every 500ms poll tick, and dispatched 2 rule-update actions per dependency (each cloning the collection mod rules array and emitting a persist diff). - memoize lookupFromDownload + findDownloadByRef identifiers per download object (WeakMap; redux state is immutable) - retain only the first (only ever consumed) lookup result per dependency - resolve dependency rules in bounded batches of 50 with DEBG progress logs - batch updateRules into a single dispatch - filter driveSelectedOptionals by session status before the per-rule findModByRef scan over all installed mods - add a 3000x3000 stress test: heap growth ~234MB -> ~10MB Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A renderer OOM during a collection install previously relaunched Vortex silently, inviting an immediate resume into the identical crash - an infinite crash loop with no visible error. The main process now writes a renderer-oom.json marker on render-process-gone with reason "oom"; on startup the collections extension consumes the marker and shows an error notification, leaving the collection paused until the user explicitly resumes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mock was a local workaround for an environment without the native toolchain; with the module built normally the tests pass without it, matching how the rest of the suite treats native deps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR addresses a renderer OOM crash loop during very large collection installs by reducing high-volume transient allocations in dependency resolution/matching, batching expensive operations, and surfacing a visible “renderer OOM” warning on next startup to avoid silent restart loops.
Changes:
- Memoize per-download lookup/identifier derivations and reduce retained dependency lookup metadata to the first result only.
- Batch dependency graph resolution and batch Redux rule updates to reduce in-flight promise chains and persist/diff churn.
- Add an OOM marker written by the main process and consumed by the collections extension on next startup to notify the user instead of silently restarting into a crash loop.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/renderer/src/extensions/mod_management/util/dependencies.ts | Adds per-download memoization, trims retained lookup results, batches dependency graph gathering, and optimizes duplicate tagging. |
| src/renderer/src/extensions/mod_management/util/dependencies.test.ts | Adds unit coverage for memoization and reference-based download resolution behavior. |
| src/renderer/src/extensions/mod_management/util/dependencies.stress.test.ts | Adds a large-scale stress test intended to guard against heap growth regressions in the matching hot path. |
| src/renderer/src/extensions/mod_management/InstallManager.ts | Reduces per-tick optional scanning cost and batches dependency rule updates into a single dispatch. |
| src/renderer/src/extensions/collections/index.ts | Reads/removes a renderer-OOM marker on startup and shows a user notification. |
| src/main/src/MainWindow.ts | Writes the renderer-OOM marker file when Electron reports render-process-gone with reason oom. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const markerPath = path.join(getVortexPath("userData"), "renderer-oom.json"); | ||
| let marker: { timestamp?: number } | undefined; | ||
| try { | ||
| marker = JSON.parse(readFileSync(markerPath, "utf8")); | ||
| unlinkSync(markerPath); | ||
| } catch { | ||
| // no marker (the usual case) or unreadable - nothing to report | ||
| return; | ||
| } |
| const before = heapUsed(); | ||
|
|
||
| let matched = 0; | ||
| for (const ref of references) { | ||
| if (findDownloadByRef(ref, downloads) !== undefined) { | ||
| ++matched; | ||
| } | ||
| } | ||
|
|
||
| const after = heapUsed(); | ||
| const growthMB = (after - before) / (1024 * 1024); | ||
|
|
||
| process.stdout.write( | ||
| `[stress] ${COUNT}x${COUNT} matching: heap growth ${growthMB.toFixed(1)} MB\n`, | ||
| ); | ||
|
|
||
| expect(matched).toBe(COUNT); | ||
| // generous bound: without memoization this run allocates hundreds of MB of | ||
| // throw-away lookup objects; with it, growth stays in the low tens of MB | ||
| expect(growthMB).toBeLessThan(192); |
|
I was able to fully install the collection that was failing for me earlier today and the previous leftover huge files from an older version also help preventing too much re-downloading. |
|
Thanks for this, and especially for the log on #23904. That log is what made the root cause findable, so this was useful even though we're closing the PR. We profiled each change against the shape from #23904 (2852 references, ~3000 local archives). Dependency resolution retains about 13.5 MB there, against a 4 GB renderer heap cap, so it isn't where the memory goes. The memoisation didn't reduce retained heap (13.5 to 13.9 MB); the batched gather didn't change it at all, since The real cause only appears on a revision update, which is what you were doing (54 to 62). Your archives carry revision 54's reference tags, so every dependency takes the tag reconciliation branch in That's also why your build stopped crashing. Batching the two dispatches into one halves the queued volume, 2635 MB to 1318 MB, which drops it under the cap. Real, but a factor of two on a quantity that grows with the square of collection size, so it moves the failure point from roughly 2850 members to about 5700 rather than removing it. The fix needs to be in the middleware instead, potentially coalescing queued operations per path so repeated writes cost the size of the state rather than size times write count. That covers every caller, not just rules. (I'm looking into it) Closing with thanks. The report and log were the valuable half, and the fix will reference them. |
Fixes #23904
Problem
Installing or updating a very large collection (reproduced with Cyberpunk 2077 p0qfwm rev 62 — 2852 required files — against a library of ~2900 existing archives) crashes the renderer with
render-process-gone {"reason":"oom"}roughly 60–120s into every attempt. Vortex silently relaunches, reports the collection incomplete, and each resume recomputes everything and crashes identically — an endless loop (resume_count: 12, ~2.3h, net progress ≈ 0 in the issue's log).Root cause
The dependency pipeline contains several
O(refs × downloads/mods)hot paths that allocate fresh objects per probe. At 2852 references × ~3000 archives that is ~8.5 million probes per pass, and several passes run repeatedly:lookupFromDownload()(util/dependencies.ts) built a fresh lookup object (plus twoSets and an identifier bundle infindDownloadByRef) for every download on every probe. Called per rule during resolution, per dependency in the post-phase ready-download scan (doInstallDependenciesPhase'sfinally, which begins right at thedone installing dependencieslog line — matching the observed crash ~30s later), and again in requeue/stall-rescue scans.pollAllPhasesCompleteticks every 500ms and calleddriveSelectedOptionals→selectedOptionalRules, which runs afindModByRefscan over all installed mods per candidate optional rule, on every tick, for the whole install. The cheap session-status filter ran after the expensive scan.lookupResults[0]is ever consumed.updateRulesdispatchedremoveModRule+addModRuleper dependency; each dispatch clones the collection mod's full 2852-entry rules array and pushes a separate diff through the persist pipeline (thepersist:diff~1/s churn in the log).The allocation rate outruns GC and the renderer heap dies. Small collections never notice because every term is small.
Changes
util/dependencies.tslookupFromDownloadand thefindDownloadByRefidentifier bundle per download object viaWeakMap(redux state objects are immutable, so object identity is a safe key).lookupResults[0]per dependency (the only entry consumers read).dependency resolution batch done {resolved, total}line so future user logs show resolution progress.tagDuplicatesskips the collateral scan for deps without a lookup result.InstallManager.tsupdateRulescollects all rule updates into onebatchDispatch(DEBGbatch updating dependency rules {count});updateModRuleshares the same logic.driveSelectedOptionalsfilters by session status (pending) before the per-rulefindModByRefscan.render-process-gonewithreason === "oom", the main process writes arenderer-oom.jsonmarker to userData (MainWindow.ts); on next startup the collections extension consumes it and shows an error notification instead of silently restarting into the same crash (collections/index.ts). The collection stays paused until explicitly resumed.Verification
util/dependencies.stress.test.ts): 3000 refs × 3000 downloads, the crash shape. Heap growth ~234 MB → ~10 MB with the memoization; suite runs ~6× faster.mod_management+collectionsrenderer suites pass (37 files / 506 tests).Not addressed (possible follow-up)
Resolution still restarts from zero on every resume (
missing: 2852each time). With this fix a re-resolution is cheap enough not to OOM, but persisting resolution progress across resumes would still save time on very large collections.🤖 Generated with Claude Code